Skip to content

chore(ci): source code tampering protection for release #2301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
May 23, 2023
Merged

chore(ci): source code tampering protection for release #2301

merged 9 commits into from
May 23, 2023

Conversation

heitorlessa
Copy link
Contributor

@heitorlessa heitorlessa commented May 22, 2023 •
edited
Loading

Issue number: #2206

Summary

Implements mechanism to seal source code, further break build job, further isolate release job with binaries only, and fixes the Git Tag creation process.

Changes

Please provide a summary of what's being changed

  • Split quality check and build jobs
  • Create integrity check
  • Create a job to seal source code after bumping version
  • Bump version and seal to prevent further boilerplate in code and workarounds
  • Cache-artifact
  • Restore-artifact and ensure it overwrites any checked out code
  • Pin down Poetry to prevent any code tampering on build + publishing
  • Update bump_version and create_tag to use sealed source code with version bumped
  • Replace subsequent checkouts to use RELEASE_COMMIT and restore artifact
  • Document every job with its details
  • Scale down Runners to use ubuntu-latest as we no longer need those 4 cores (savings!!)
  • Lower permissions further as jobs are isolated
  • Add simple debug to display pyproject.toml in case sealed source code is incorrect
  • Forcefully fail release if sealed source code hasn't been found or uploaded correctly
  • Test new release
  • Test whether tag created contains latest source code with version bumped
  • Document overall process at the top
  • Verify integrity hash (source and build) in key steps

Additional fixes

  • bump_version needs to install poetry
  • Fix create_tag as it needs to add staged changes before pushing... otherwise it pushes the current commit
  • No longer need custom build machines for build/release.. but layer only

Threats to address

  • Checking out newer code during the release process
    • Forcefully use RELEASE_COMMIT during checkout
    • Verifies hash throughout jobs (source and build)
  • Source code tampering from dependencies or workflows
    • seal job to hash and reuse artifact throughout release
    • pipx install git+<url+SHA> for poetry to prevent rogue package manager
  • Tamper build artifact during release
    • release-<build-hash> for traceability and immutability
    • pipx install git+<url+SHA> for poetry to prevent rogue package manager
    • release job doesn't install any dependencies but a pinned poetry
  • Tamper git tag during release
    • seal job to hash and reuse artifact
    • No dependencies installed
    • Forcefully add pyproject.toml that changed in seal to our local git tree, then push that with the new tag only. This also solves the data race condition where PR is only merged later and our tag uses an older version.
  • Push modified code to repository via 3rd-party actions / dependencies
    • Protected branch enabled in trunk
    • All source code modifications generate PR and require maintainer's review
    • Least privilege permissions are enforced per job
    • For Git tags, source code tampering check is also enforced
  • Secret ex-filtration 3rd-party action / dependencies
    • GitHub Environment Secrets are used over standard Secrets
    • Encrypted secrets are only available for release environment
    • release environment requires manual approval before proceeding
  • Release credentials overtake
    • Setup PyPi Trusted Publisher
    • Release credentials are temporary and only issued to what's configured in PyPi Trusted Publisher (org+repo+workflow+environment name)

User experience

Please share what the user experience looks like before and after this change

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Is this a breaking change?

RFC issue number:

Checklist:

  • Migration process documented
  • Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@heitorlessa
chore: create download-artifact, upload-artifact
7c25cf1
@heitorlessa
chore: seals source code, separate quality check & build
8663cee 
Uses new mechanism to seal source code, run quality checks separately,
and always reuses sealed source code in every step to prevent source
code tampering.

It documents every job purpose, and creates a new faster action
to upload and download artifacts.
@heitorlessa heitorlessa requested a review from a team as a code owner May 22, 2023 10:10
@heitorlessa heitorlessa requested review from rubenfonseca and removed request for a team May 22, 2023 10:10
@boring-cyborg boring-cyborg bot added the github-actions Pull requests that update Github_actions code label May 22, 2023
@pull-request-size pull-request-size bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 22, 2023
@github-actions github-actions bot added the internal Maintenance changes label May 22, 2023
@rubenfonseca
Copy link
Contributor

Reviewing now

rubenfonseca
rubenfonseca suggested changes May 22, 2023
View reviewed changes
Copy link
Contributor

@rubenfonseca rubenfonseca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super good job, TIL that you can have your own custom actions on the same repo. Just left some questions and a tiny comment.

.github/actions/download-artifact/action.yml
# NOTES
#
# Upload-artifact and download-artifact takes ~2m40s to upload 8MB
# so this is custom action cuts down the entire operation to 1s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@heitorlessa @rubenfonseca
chore: ruben's feedback
3da540a 
Co-authored-by: Ruben Fonseca <[email protected]>
Signed-off-by: Heitor Lessa <[email protected]>
rubenfonseca
rubenfonseca approved these changes May 22, 2023
View reviewed changes
Copy link
Contributor

@rubenfonseca rubenfonseca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing!

@heitorlessa heitorlessa changed the title chore: source code tampering protection for release chore(ci): source code tampering protection for release May 22, 2023
@heitorlessa
Copy link
Contributor Author

@rubenfonseca gonna need your review one more time as I've added a hash verifier as per @leandrodamascena ask. Leandro reviewed

@heitorlessa
Copy link
Contributor Author

heitorlessa commented May 23, 2023 •
edited
Loading

Successful run publishing to Test PyPi repo (minus layers for good reasons and no change there): https://github.com/heitorlessa/aws-lambda-powertools-test/actions/runs/5053853611

leandrodamascena
leandrodamascena approved these changes May 23, 2023
View reviewed changes
Copy link
Contributor

@leandrodamascena leandrodamascena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVED!! Amazing work @heitorlessa!

@leandrodamascena leandrodamascena merged commit 1995d0f into aws-powertools:develop May 23, 2023
@heitorlessa
Copy link
Contributor Author

Aleluia! <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rubenfonseca rubenfonseca rubenfonseca approved these changes

@leandrodamascena leandrodamascena leandrodamascena approved these changes

Assignees
No one assigned
Labels
github-actions Pull requests that update Github_actions code internal Maintenance changes size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@heitorlessa @rubenfonseca @leandrodamascena